Skip to content

Locations performance improvements#14718

Open
dogboat wants to merge 47 commits intoDefectDojo:devfrom
dogboat:locations-perf-improvements
Open

Locations performance improvements#14718
dogboat wants to merge 47 commits intoDefectDojo:devfrom
dogboat:locations-perf-improvements

Conversation

@dogboat
Copy link
Copy Markdown
Contributor

@dogboat dogboat commented Apr 21, 2026

Rework importing when using Locations to be not as slow.

Some changes are made to AbstractLocation and the URL model to support this. First, the identity_hash attribute and set_identity_hash() method are moved from URL to AbstractLocation to support bulk operations across location types. AbstractLocation now calls set_identity_hash() as part of its clean() routine. Finally, a bulk_get_or_create() classmethod is added to AbstractLocation; it is responsible for (hopefully efficiently) bulk getting-or-creating an iterable of the appropriate type (e.g., URLs). It works by querying for existing locations based on identity_hash values, and only creating new ones using Django's bulk_create() method.

Both LocationManager and EndpointManager have been changed. Of greatest impact, LocationManager has been updated to use the same accumulator pattern that EndpointManager was recently updated to use (thanks Val!). While doing this, I noticed the opportunity to consolidate things a bit; rather than maintaining the V3 if/else structure throughout, I standardized around the set of methods used by EndpointManager. At the same time, I inserted another level of indirection -- LocationHandler, which under the covers delegates to one or the other manager -- to help keep the two managers more in line going forward. The intent is, we won't be able to introduce one-off features for only Endpoints or Locations without at least considering how to handle the other. Plus, it removes a lot of the "#TODO: DELETE ME IN THE FUTURE MAYBE" comments right now. Additionally, LocationProductReference status is now kept in sync with finding-level status changes during reimport (if any finding ref is Active, the product ref is Active; otherwise Mitigated). This was a pre-existing gap.

The general process was, take the excellent pattern set by EndpointManager, and beat it until it worked with Locations too.

In my very unscientific testing of a 3k findings import, V2 finished in ~21s, while V3 finished in ~23s, putting Locations solidly within tickling distance of Endpoints.

@github-actions github-actions Bot added New Migration Adding a new migration file. Take care when merging. unittests labels Apr 21, 2026
@mtesauro mtesauro added this to the 2.57.3 milestone Apr 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

New Migration Adding a new migration file. Take care when merging. unittests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants